1 link tagged with all of: github + replica lag + connection pooling + long timeouts
Links
GitHub's September 13 incident stemmed from a background data-cleanup job that overwhelmed a shared database cluster because the safeguard only monitored replica lag, not the primary's connection limit. Long request timeouts and retry loops made recovery harder and exposed how offline jobs can silently tank online systems.
- The safeguard watched the wrong metric (replica lag stayed healthy while the primary hit its connection limit), showing that health signals must track the actual bottleneck, not just a proxy.
- Long timeouts are invisible failures: they don't surface during normal operation like short timeouts do, but instead accumulate in-flight requests that starve the system of resources.
- Offline jobs against shared online databases need rate-limiting and load-shedding controls built in advance—you can't improvise this during an incident when you're already saturated.
- The fixes (rate-limits, better monitoring, request timeouts, database sharding) all add complexity, which is the eternal tradeoff of preventing incidents.
github
connection pooling
replica lag
long timeouts